Image-capturing device

ABSTRACT

Because conversion from an invisible wavelength band to a visible wavelength band has been performed arbitrarily on a device-by-device basis, the color gamut in the visible color space could not be utilized sufficiently. A first aspect of the present invention provides an image-capturing device including: an image-capturing unit that is photosensitive to light in an invisible band; a generating unit that generates invisible wavelength information defined based on a sensitivity characteristic, which is an output characteristic of the image-capturing unit, to an object light flux wavelength for conversion of image-capturing data generated from an output signal of the image-capturing unit into visible color space image data; and a processing unit that relates the invisible wavelength information to the image-capturing data.

The contents of the following Japanese and International patentapplications are incorporated herein by reference:

-   -   NO. 2015-72660 filed on Mar. 31, 2015, and    -   NO. PCT/JP2016/060571 filed on Mar. 30, 2016.

BACKGROUND 1. Technical Field

The present invention relates to an image-capturing device.

2. Related Art

An image-capturing system that allocates the visible three primarycolors (RGB) to invisible three wavelength bands having mutuallydifferent central wavelengths, respectively, has been known.

PRIOR ART DOCUMENT Patent Document

[Patent Document 1] WO2007/083437

Because conversion from an invisible wavelength band into a visiblewavelength band has been performed arbitrarily on a device-by-devicebasis, the color gamut in the visible color space could not be utilizedsufficiently.

SUMMARY

One aspect of the present invention provides an image-capturing deviceincluding: an image-capturing unit that is photosensitive to light in aninvisible band; a generating unit that generates invisible wavelengthinformation defined based on a sensitivity characteristic, which is anoutput characteristic of the image-capturing unit, to an object lightflux wavelength for conversion of image-capturing data generated from anoutput signal of the image-capturing unit into visible color space imagedata; and a processing unit that relates the invisible wavelengthinformation to the image-capturing data.

The summary clause does not necessarily describe all necessary featuresof the embodiments of the present invention. The present invention mayalso be a sub-combination of the features described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a figure for explaining a configuration of a digital camera.

FIG. 2 is a figure for explaining a bandpass filter arranged on eachphotoelectric converting unit of an image sensor.

FIG. 3 is a figure for explaining an output characteristic of an imagesensor.

FIG. 4 is a figure for explaining a space defined by an outputcharacteristic of an image sensor.

FIG. 5 is a figure for explaining a correspondence between animage-capturing system color space and a display system color space.

FIG. 6 is a figure for explaining a color distribution of a displaysystem color space.

FIG. 7 is a figure for explaining a reference point.

FIG. 8 is a figure showing a file structure of an image-capturing file.

FIG. 9 is a figure for explaining a relationship between the type ofbandpass filter and an image-capturing system color space.

FIG. 10 is a figure for explaining a relationship between bandpassfilters and object spectra.

FIG. 11 is a flow diagram showing a process flow of a digital camera.

FIG. 12 is a figure showing one example of image-capturing system colorspace information.

FIG. 13 is a figure for explaining a correspondence between animage-capturing system color space and a display system color space.

FIG. 14 is a figure for explaining one example of color allocation.

FIG. 15 is a figure for explaining another example of color allocation.

FIG. 16 is a figure for explaining another example of color allocation.

FIG. 17 is a flow diagram showing a process flow of a digital camera.

FIG. 18 is a figure for explaining another example of a correspondencebetween an image-capturing system color space and a display system colorspace.

FIG. 19 is a figure for explaining another example of a correspondencebetween an image-capturing system color space and a display system colorspace.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, (some) embodiment(s) of the present invention will bedescribed. The embodiment(s) do(es) not limit the invention according tothe claims, and all the combinations of the features described in theembodiment(s) are not necessarily essential to means provided by aspectsof the invention.

FIG. 1 is a figure for explaining a configuration of a digital camera 10as one example of an image-capturing device according to the presentembodiment. The digital camera 10 is an image-capturing device thatcaptures an image of an invisible band object light flux. The digitalcamera 10 includes a taking lens 20 as a photographing optical system,and an image sensor 100. The taking lens 20 guides, to the image sensor100, an object light flux that enters along an optical axis 21. Inaddition to the taking lens 20 and the image sensor 100, the digitalcamera 10 includes a control unit 201, an A/D conversion circuit 202, awork memory 203, a drive unit 204, an image processing unit 205, asystem memory 206, a memory card IF 207, a manipulating unit 208, adisplay unit 209, an LCD drive circuit 210 and a communicating unit 211.

As illustrated in the figure, the direction parallel with the opticalaxis 21 that points to the image sensor 100 is determined as the Z-axispositive direction, the direction that points into the paper surface ona plane orthogonal to the Z-axis is determined as the X-axis positivedirection, and the direction that points upward on the paper surface isdefined as the Y-axis positive direction. Regarding a relationship withthe composition in photographing, the X-axis becomes the horizontaldirection, and the Y-axis becomes the vertical direction. In some of thefigures that follow, coordinate axes are displayed with the coordinateaxes of FIG. 1 being used as the reference so that the orientation ofeach figure can be known.

The taking lens 20 is configured with a plurality of optical lensgroups, and forms an image of an object light flux from a scene in theproximity of its focal plane. The taking lens 20 may be a replaceablelens that can be attached to and detached from the digital camera 10. Inthis case, the camera body functions as an image-capturing device. Forconvenience of explanation, in FIG. 1, the taking lens 20 isrepresentatively represented with a single virtual lens arranged in theproximity of the pupil.

The image sensor 100 is arranged in the proximity of the focal plane ofthe taking lens 20. The image sensor 100 is an infrared image sensorphotosensitive to light in an invisible band. In the present embodiment,as one example of it, the image sensor 100 is photosensitive to light inthe range of 800 nm to 2000 nm in the near infrared band of 800 nm to2500 nm. These near infrared band and range of photosensitivity are notlimited to those in the present example.

The image sensor 100 includes a plurality of pixels arrayedtwo-dimensionally. Each one among the plurality of pixels includes aphotoelectric converting unit, and a bandpass filter providedcorresponding to the photoelectric converting unit. As described indetail below, in the present embodiment, there are three types ofbandpass filter, and each one among the plurality of photoelectricconverting units is provided with one of the bandpass filters.

The image sensor 100 is timing-controlled by the drive unit 204 toconvert an object image formed on its light receiving surface into apixel signal and output it to the A/D conversion circuit 202. The A/Dconversion circuit 202 converts the pixel signal as an output signaloutput by the image sensor 100 into a digital signal. It outputsimage-capturing data obtained by the digital conversion to the workmemory 203.

The control unit 201 plays a role as a generating unit that generatesinfrared wavelength information as one example of invisible wavelengthinformation for converting image-capturing data generated from theoutput signal of the image sensor 100 into visible color space imagedata. The visible color space shows a range of colors that apredetermined color coordinate system can handle. In other words,because it shows a range of colors that a color coordinate system canhandle, the visible color space is in some cases referred to as adisplay system color space in the present specification. As described indetail below, the infrared wavelength information is defined based onthe sensitivity characteristic which is the output characteristic of theimage sensor 100 to an object light flux wavelength.

The image processing unit 205 performs various types of processing suchas brightness correction processing on image-capturing data using thework memory 203 as a workspace. Also, the image processing unit 205plays a role as a processing unit that relates infrared wavelengthinformation as tag information to the image-capturing data that hasundergone the various types of processing. An image-capturing file inwhich the image-capturing data and the infrared wavelength informationare related to each other is recorded in a memory card 220 attached tothe memory card IF 207.

The control unit 201 and the image processing unit 205 cooperate witheach other to convert the image-capturing data into visible color spaceimage data using the infrared wavelength information. Details ofconversion are described below.

The image processing unit 205 can convert infrared wavelength band-basedimage-capturing data into visible wavelength band-based image data togenerate the image data by allocating mutually different visible bandsto respective wavelength bands of three types of bandpass filter.

The generated image data is converted into a display signal by the LCDdrive circuit 210, and displayed on the display unit 209. As describedin detail below, even objects that are difficult to color-discriminatein a visible wavelength band-based image, like water and oil forexample, can be color-discriminated in an infrared wavelength band-basedimage. A menu screen for various types of setting is also displayed onthe display unit 209. For example, a menu screen related to setting of areference point described below is displayed. Also, the generated imagedata is recorded in the memory card 220 attached to the memory card IF207.

The system memory 206 records a program to control the digital camera10, various types of parameter and the like. In the present embodiment,an image-capturing parameter is stored. The image-capturing parameterincludes information indicative of a light source characteristic,information indicative of transmittances of three types of bandpassfilter, information indicative of the photosensitivity of the imagesensor 100 and the like. The information indicative of transmittance maybe stored as a table in which transmittances are respectively associatedwith wavelength bands of constant intervals, or may be stored as afunction for calculating a transmittance corresponding to a wavelengthband.

The manipulating unit 208 accepts a manipulation of a user to output amanipulation signal corresponding to the manipulation to the controlunit 201. For example, if a menu screen related to setting of areference point is displayed on the display unit 209, it outputs amanipulation signal related to the setting of the reference point to thecontrol unit 201 according to the manipulation. A user can select amethod of setting a reference point or the like through the manipulatingunit 208.

Also, the manipulating unit 208 includes manipulating member such as arelease switch, a cross-key or an OK key. The release switch isconfigured with a press-button that can sense manipulations at two stepsin its pressing direction. The control unit 201 performs AF, AE and thelike which are photographing preparatory operations by sensing SW1 whichis a first step pressing, and performs an object image acquiringoperation by the image sensor 100 by sensing SW2 which is a second steppressing. In the present embodiment, AF is performed so that an objectimage is focused in the infrared wavelength band.

The communicating unit 211 communicates with another device. Thecommunicating unit 211 transmits an image-capturing file to the otherdevice according to a manipulation by a user through the manipulatingunit 208. Examples of the other device include a device provided with adisplay unit such as a personal computer, a smartphone or a tablet, anda server device on the Internet, and the like.

A series of photographing sequences is started upon the manipulatingunit 208 accepting a manipulation by a user and outputting amanipulation signal to the control unit 201.

FIG. 2 is an explanatory figure for a bandpass filter arranged on eachphotoelectric converting unit of the image sensor 100. Each one amongthe three types of bandpass filter allows part of continuous nearinfrared bands in an object light flux to pass therethrough. The bandpassage of which is allowed is mutually different among the three typesof bandpass filter. As illustrated in the figure, in the presentembodiment, a NIR1 filter, a NIR2 filter and a NIR3 filter are providedas the three types of bandpass filter. The NIR1 filter, the NIR2 filtersand the NIR3 filter are allocated like a Bayer array for four pixels 101consisting of two pixels×two pixels. In more detail, the NIR2 filtersare allocated to two upper left and lower right pixels, the NIR1 filteris allocated to one lower left pixel and the NIR3 filter is allocated toone upper right pixel. The arrangement of the bandpass filters is notlimited to that in the present example.

In the image sensor 100 as a whole, each one among a plurality of pixelsarrayed two-dimensionally is provided discretely with any of the NIR1filter, the NIR2 filter and the NIR3 filter. Accordingly, it can be saidthat the image sensor 100 detects an incoming object light flux whileseparating it into respective wavelength bands. In other words, theimage sensor 100 performs photoelectric conversion while separating anobject image formed on its light receiving surface into three wavelengthbands that are in the infrared band and are mutually different.

FIG. 3 is a figure for explaining an output characteristic of the imagesensor 100. (a) of FIG. 3 is a figure for explaining bandpass filters.The horizontal axis indicates wavelength [nm], and the vertical axisindicates transmittance [%]. (b) of FIG. 3 is a figure for explainingthe photosensitivity of the image sensor 100. The horizontal axisindicates wavelength [nm], and the vertical axis indicatesphotosensitivity. Here, the photosensitivity is normalized assuming thatthe photosensitivity for the highest sensitivity wavelength is 100. Inorder to simplify the explanation, the conversion efficiency is assumedas being 100% if the photosensitivity is 100. (c) of FIG. 3 is a figureshowing an output characteristic determined by the photosensitivities ofthe bandpass filters and the image sensor 100. The horizontal axisindicates wavelength [nm], and the vertical axis indicatesphotosensitivity. The broken lines indicate the transmittancedistributions of the NIR1 filter, the NIR2 filter and the NIR3 filter,and the chain line indicates the photosensitivity distribution of theimage sensor 100. The solid lines indicate the sensitivity distributionsof output characteristics.

As shown in (a) of FIG. 3, the respective transmittance distributionshapes of the NIR1 filter, the NIR2 filter and the NIR3 filter areapproximately the same overall. In more detail, the NIR1 filter istransmissive to about 700 nm to about 1550 nm, and the peak wavelengthλa of the NIR1 filter is 1150 nm. The NIR2 filter is transmissive toabout 950 nm to about 1800 nm, and the peak wavelength kb of the NIR2filter is 1400 nm. The NIR3 filter is transmissive to about 1250 nm toabout 2100 nm, and the peak wavelength λc of the NIR3 filter is 1650 nm.

Each of the NIR1 filter, the NIR2 filter and the NIR3 filter istransmissive to the peak wavelengths of the other band filters.Specifically, the NIR1 filter is transmissive to the peak wavelength ofthe NIR2 filter. Likewise, the NIR3 filter is transmissive to the peakwavelength of the NIR2 filter. Also, the NIR2 filter is transmissive tothe respective peak wavelengths of the NIR1 filter and the NIR3 filter.

As shown in (b) of FIG. 3, the image sensor 100 is photosensitive tolight in the wavelength band of about 800 nm to about 2000 nm. In moredetail, the photosensitivity rapidly increases in the wavelength band ofabout 800 nm to the about 1050 nm. The photosensitivity is 100 in therange of about 1050 nm to about 1750 nm. The photosensitivity rapidlydecreases in the range of about 1750 nm to about 2000 nm.

Here, the output characteristic of the image sensor 100 is calculated bymultiplication of the transmittances of bandpass filters and thephotosensitivity of the image sensor 100. In the present embodiment, theoutput characteristic of the image sensor 100 is defined as a range overwhich the image sensor 100 has a photosensitivity equal to or higherthan a predetermined threshold (for example, 1%) in the photosensitivitycalculated by multiplication.

As shown in (c) of FIG. 3, because in the range of about 1050 nm toabout 1750 nm, the photosensitivity of the image sensor 100 is 100, thetransmittances of the NIR1 filter, the NIR2 filter and the NIR3 filterare an output characteristic as is. On the other hand, because in therange of about 800 nm to about 1050 nm and the range of about 1750 nm toabout 2000 nm, the photosensitivity of the image sensor 100 is not 100,the transmittances of the NIR1 filter, the NIR2 filter and the NIR3filter are not an output characteristic as is. In the range of about 800nm to about 1050 nm, the output characteristic is calculated bymultiplying the respective transmittances of the NIR1 filter and theNIR2 filter by the photosensitivity of the image sensor 100. Likewise,in the range of about 1750 nm to about 2000 nm, it is calculated bymultiplying the respective transmittances of the NIR2 filter and theNIR3 filter by the photosensitivity of the image sensor 100. As aresult, the lower limit value of the output characteristic is 850 nm,and its upper limit value is 1950 nm.

FIG. 4 is a figure for explaining a space defined by the outputcharacteristic of the image sensor 100. In the present embodiment, avirtual color coordinate system corresponding to a visible CIE colorcoordinate system is defined in order to handle the range of the outputcharacteristic, that is, the invisible wavelength band in a similarmanner to the visible wavelength band. Specifically, a region 301defined by the lower limit value and upper limit value of the outputcharacteristic (a region surrounded by the dotted line) is defined as aregion corresponding to a visible region in a chromaticity diagramdescribed below. That is, the curved part of the region 301 is definedby continuous wavelengths from the lower limit value, 850 nm, to theupper limit value, 1950 nm. Because the visible region is generallyexpressed with a horseshoe shape, the region 301 can be made correspondto the entire visible region by shaping the region 301 into a similarfigure to the visible region. By defining the region 301 in this manner,if the region 301 is converted into the visible region as describedbelow, the entire region 301 can be included in the visible region, andthe entire visible region can be utilized. FIG. 4 illustrates x′, y′corresponding to a chromaticity coordinate, x, y as coordinate axes. Thehorizontal axis is x′, and the vertical axis is y′. Although in thepresent embodiment, the region 301 is expressed in a horseshoe shape inimitation of the visible region, the shape is not limited to a horseshoeshape. The region 301 may have any shape as long as it includes atriangle indicated by a region 303 described below.

If the region 301 is defined, a coordinate (m, n) of a reference pointRefP is set subsequently. As described in detail below, in the presentembodiment, the coordinate (m, n) of the reference point RefP is set, asa default position, at a position corresponding to a reference whitedescribed below. If the lower limit value, the upper limit value and thereference point RefP are determined, a relationship between thereference point RefP and a wavelength at each point of the curved partof the region 301 is determined so as to set the coordinate (m, n) ofthe reference point RefP to (0.33, 0.33). That is, the coordinates ofthe peak wavelength λa, the peak wavelength kb and the peak wavelengthλc are uniquely determined.

A region 302 is a region formed by linking, with straight lines, therespective points of the peak wavelength λa, the peak wavelength kb andthe peak wavelength λc. The region 302 is larger than a region indicatedby a region 303 described below.

The region 303 is a region formed by linking, with straight lines, apoint AB on a straight line linking the peak wavelength λa and thereference point RefP, a point CD on a straight line linking the peakwavelength λb and the reference point RefP and a point EF on a straightline linking the peak wavelength λc and the reference point RefP. Theregion 303 indicates a color space that is actually reproducible.Although because the region 303 is a region of the infrared wavelengthband, it is actually a region that cannot be visually recognized incolors, it is referred as a color space for convenience of explanationin the present specification. In the present specification, the region303 is referred to as an image-capturing system color space as a spacecorresponding to the display system color space. Also, the region 303 isin some cases referred to as a virtual color space in a correspondingmanner to the visible color space. In the image-capturing system colorspace, an infrared wavelength band passage of which each bandpass filterhas allowed is expressed by a combination of numerical values. Theregion 303 is determined by the transmittance distribution of eachbandpass filter.

If the coordinate (m, n) of the reference point RefP is set, a displayresolution is determined. In other words, distances from the respectivepoints of the peak wavelength λa, the peak wavelength kb and the peakwavelength λc to respective corresponding vertexes of the region 303 aredetermined. That is, the coordinate (a, b) of the point AB, thecoordinate (c, d) of the point CD and the coordinate (e, f) of the pointEF are determined. Thereby, the area of the region 303, that is, thecolor-saturation, is determined.

FIG. 5 is a figure for explaining a correspondence between animage-capturing system color space and a display system color space. (a)of FIG. 5 is the same as FIG. 4. (b) of FIG. 5 shows a visible colorcoordinate system chromaticity diagram. The horizontal axis is x, andthe vertical axis is y.

As mentioned above, the region 301 in (a) of FIG. 5 is associated with aregion 401 indicating a visible region in (b) of FIG. 5. Specifically,850 nm which is the lower limit value of the region 301 and 380 nm whichis the lower limit value of the region 401 are associated with eachother, and 1950 nm which is the upper limit value of the region 301 and700 nm which is the upper limit value of the region 401 are associatedwith each other. The widths between wavelengths of respective points onthe curved part of the region 401 in (b) of FIG. 5 are not constant. Therelationship between the reference white RefW and the wavelengths of therespective points on the curved part of the region 401 is determined toset the coordinate (o, p) of the reference white RefW to (0.33, 0.33).Specifically, the widths between wavelengths are set to be small in therange of 380 nm to 470 nm, large in the range of 470 nm to 620 nm, andsmall in the range of 620 nm to 700 nm. Likewise, the widths betweenwavelengths of respective points in the curved part of the region 301 in(a) of FIG. 5 are set to be small around the lower limit value andaround the upper limit value, and large in the other portions. Becausethe lower limit value and upper limit value of the region 401 aredetermined corresponding to the human visual characteristics, and thusare fixed. On the other hand, as described in detail below, the lowerlimit value and upper limit value of the region 301 are in the infraredwavelength band, determined irrespective of the human visualcharacteristics, and thus are variable.

In the present embodiment, the coordinate (m, n) of the reference pointRefP is associated with the coordinate (o, p) of the reference whiteRefW as a particular coordinate. The region 303 in (a) of FIG. 5 (thatis, the virtual color space) is associated with a region 403 in (b) ofFIG. 5 (that is, the visible color space). Specifically, the coordinate(a, b) of the vertex AB is associated with the coordinate (g, h) of thevertex GH. That is, coordinates around the coordinate (a, b) of thevertex AB are associated with bluish colors. The coordinate (c, d) ofthe vertex CD is associated with the coordinate (i, j) of the vertex IJ.That is, coordinates around the coordinate (c, d) of the vertex CD areassociated with greenish colors. The coordinate (e, f) of the vertex EFis associated with the coordinate (k, l) of the vertex KL. That is,coordinates around the coordinate (e, f) of the vertex EF are associatedwith reddish colors. If coordinates representing an object are mapped tothe region 303, the coordinates can be converted into coordinates in theregion 403. That is, they can be converted into visible colors. Becausethe vertex AB of the region 303 is positioned on a line between thereference point RefP and the peak wavelength λa, the proportion of astraight line linking the peak wavelength λa and the vertex AB to astraight line linking the reference point RefP and the peak wavelengthλa is determined. It can be expressed at a similar proportion also inthe visible color coordinate system chromaticity diagram. That is, ifthe vertex AB of the region 303 and the peak wavelength λa aredetermined, the coordinate (g, h) of the vertex GH can be determined.Likewise, the coordinate (i, j) of the vertex IJ and the coordinate (k,l) of the vertex KL can be determined.

FIG. 6 is a figure for explaining a color distribution of a displaysystem color space. Similar to (b) of FIG. 5, FIG. 6 shows a visiblecolor coordinate system chromaticity diagram. The horizontal axis is x,and the vertical axis is y. As mentioned above, the region 403corresponds to a visible color space. Although colors shift continuouslyin the visible color space, it is assumed in the explanation that theregion 403 is divided into three regions of “bluish,”, “greenish,” and“reddish” as indicated by broken lines to simplify the explanation.

FIG. 7 is a figure for explaining the reference point RefP. (a) of FIG.7 is a figure for explaining object spectra. The horizontal axisindicates wavelength [nm], and the vertical axis indicates normalizedspectral intensity. Objects are oil and water, as examples. The solidline indicates a spectrum of oil, and the broken line indicates aspectrum of water.

As illustrated in the figure, oil has a spectral intensity in a widerange between 900 nm and 1700 nm. In more detail, in the range of 900 nmto around 1350 nm, the spectral intensity of oil somewhat decreases inthe range between around 1150 nm to around 1200 nm, but increasesoverall, and increases from around 0.1 to around 1. In the range ofaround 1350 nm to around 1700 nm, it temporarily increases in the rangebetween around 1410 nm to around 1480 nm, but decreases overall, anddecreases from around 1 to around 0.1.

The spectral intensity of water increases from a little less than 0.2 toaround 1 in the range of 900 nm to around 1100 nm. In the range ofaround 1100 nm to around 1400 nm, it temporarily increases in the rangeof around 1150 nm to around 1270 nm, but decreases overall, anddecreases from around 1 to around 0.2. In the range of around 1400 nm toaround 1700 nm, it stays approximately flat, and somewhat decreases inthe range of around 1650 nm to 1700.

Because the spectral intensity of oil ranges over a relatively widerange of the near infrared wavelength band, oil is recognized in whitishcolors as compared with water in an image after conversion into thevisible wavelength band. On the other hand, because the spectralintensity of water increases rapidly in the range up to around 1100 nm,and decreases rapidly in the range after around 1250 nm, it differssignificantly from the spectral intensity of oil in the band up toaround 1100 nm and the band after around 1200 nm. Because the spectralintensity corresponding to the wavelength bands of the NIR2 filter andthe NIR3 filter is relatively low, water is recognized in bluish colorsoverall in an image after conversion into the visible wavelength band.

(b), (c), (d) and (e) of FIG. 7 are figures for explaining the positionof the reference point RefP. Here, for simplification of the figure,only the virtual color space is extracted and illustrated. In eachfigure, a region 501 indicates a region in which water is mapped to thevirtual color space, and a region 502 indicates a region in which oil ismapped to the virtual color space. As mentioned above, the referencepoint RefP is associated with the reference white Ref W. The colorallocated to water and the color allocated to oil are determined by arelative positional relationship between the reference point RefP, andthe region 501 and the region 502.

In the example of (b) of FIG. 7, the reference point RefP is set at thesame position as that of the reference point RefP shown in FIG. 4 andFIG. 5. In this case, as mentioned above, water is recognized in bluishcolors, and oil is recognized in whitish colors.

In the example of (c) of FIG. 7, the reference point RefP is set closerto the region 501 than the position of the reference point RefP shown in(b) of FIG. 7 is. Because the relative distance between the referencepoint RefP and the region 501 becomes shorter, water is recognized notin bluish colors but in whitish colors. On the other hand, oil isrecognized in dark yellow colors rather than in whitish colors. This isbecause the reference point RefP is set closer to the region 501, thatis, closer to blue colors, thereby increasing the relative distancebetween the reference point RefP and the region 502.

In the example of (d) of FIG. 7, the reference point RefP is set closerto the region 502 than the position of the reference point RefP shown in(b) of FIG. 7 is. Because the relative distance between the referencepoint RefP and the region 501 becomes longer, water is recognized indarker blue colors. On the other hand, oil is recognized in colors closeto white colors. This is because the reference point RefP is set closerto the region 502, that is, closer to red colors, thereby decreasing therelative distance between the reference point RefP and the region 502.

In the example of (e) of FIG. 7, the reference point RefP is set closerto the lower side on the paper surface than the position of thereference point RefP shown in (b) of FIG. 7 is. Because in this case,the hue changes, water is recognized in blue-greenish colors, and oil isrecognized in yellow-greenish colors.

As described above, colors of objects after conversion into the visiblewavelength band can be adjusted by the position of the reference pointRefP. By setting the position of the reference point RefP correspondingto an object spectrum, an object can be color-discriminated easily. Ifcolor difference discrimination thresholds such as those of a McAdamellipse are stored in the system memory 206 in advance, the position ofthe reference point RefP may be set by referring to the discriminationthresholds so as to make color-discrimination easy. Although in theexplanation above, an example of color-discrimination of water and oilwas given, the reference point RefP may be set in the region 501 ifforeign substances mixed into water are to be discriminated.

(f), (g), (h) and (i) of FIG. 7 show virtual color distributionsallocated to the virtual color space. The visible color space colordistributions are determined according to the human visualcharacteristics. Accordingly, the visible color space colordistributions are fixed regardless of changes in the position of thereference value RefP. If the position of the reference value RefPchanges, a correspondence between virtual color space wavelengths andvisible color space wavelengths changes. For example, although if thereference value RefP is set at the position shown in (b) of FIG. 7, 1150nm and 470 nm correspond to each other, if the reference value RefP isset at the position shown in (c) of FIG. 7, 1300 nm and 470 nmcorrespond to each other. That is, a wider band range in the infraredwavelength band is associated with bluish colors. In other words, thismeans that if a color distribution is tentatively allocated to thevirtual color space, the region of bluish colors becomes larger. In viewof this, here, a correspondence between virtual color space wavelengthsand visible color space wavelengths is explained by tentativelyallocating a color distribution to the virtual color space.

(f) of FIG. 7 shows a color distribution in the case of the referencevalue RefP in (b) of FIG. 7. In this case, because the position of thereference value RefP and the position of the reference white RefWcorrespond to each other, the color distribution becomes the same as thecolor distribution in the visible space shown in FIG. 6.

(g) of FIG. 7 shows a color distribution in the case of the referencevalue RefP in (c) of FIG. 7. In this case, because the reference pointRefP is set closer to the region 501, in the virtual color space, theregion positioned on the left side of the paper surface from thereference point RefP becomes smaller, and conversely the regionpositioned on the right side on the paper surface becomes larger.Because the position of the reference white RefW corresponding to thereference point RefP is fixed, in the visible color space, the smallregion positioned on the left side in the virtual color space isallocated to a relatively large region. Conversely, the large regionpositioned on the right side in the virtual color space is allocated toa relatively small region. This means that in a color distribution forthe virtual color space, the region of bluish colors becomes larger, andconversely the regions of reddish colors and greenish colors becomesmaller. That is, by the reference point RefP being set closer to theregion 501, variations in bluish color-expression can be increased.

(h) of FIG. 7 shows a color distribution in the case of the referencevalue RefP in (d) of FIG. 7. In this case, because the reference pointRefP is set closer to the region 502, in the virtual color space, theregion positioned on the left side of the paper surface from thereference point RefP becomes larger, and conversely the regionpositioned on the right side on the paper surface becomes smaller.Accordingly, in the visible color space, a large region positioned onthe left side in the virtual color space is allocated to a relativelysmall region. Conversely, the small region positioned on the right sidein the virtual color space is allocated to a relatively large region.This means that in a color distribution for the virtual color space, theregions of bluish colors and greenish colors become smaller, andconversely the region of reddish colors becomes larger. That is, by thereference point RefP being set closer to the region 502, variations inreddish color-expression can be increased.

(i) of FIG. 7 shows a color distribution in the case of the referencevalue RefP in (e) of FIG. 7. In this case, because the reference pointRefP is set closer to the lower side on the paper surface, in thevirtual color space, the region positioned on the upper side of thepaper surface from the reference point RefP becomes larger, andconversely the region positioned on the lower side on the paper surfacebecomes smaller. Accordingly, in the visible color space, the largeregion positioned on the upper side in the virtual color space isallocated to a relatively small region. Conversely, the small regionpositioned on the lower side in the virtual color space is allocated toa relatively large region. This means that in a color distribution forthe virtual color space, the region of greenish colors becomes smaller,and conversely the regions of bluish colors and reddish colors becomelarger. That is, by the reference point RefP being set closer to thelower side on the paper surface, variations in bluish color- and reddishcolor-expression can be increased.

FIG. 8 is a figure showing a file structure of an image-capturing file.As mentioned above, an image-capturing file has a file structureconsisting, as its main elements, of main image-capturing data which isimage-capturing data itself, and infrared wavelength information whichis tag information of the image-capturing data.

As illustrated, in the infrared wavelength information, various types ofinformation are written while being classified into respectivecategories. In the following, main types of information are explained.

In the category of file information, a type, size, image information andthe like are written. Specifically, “JPG image” is written as the type,and “4.09 MB” is written as the size. As the image information, thenumbers of dots in the x direction and the y direction of the presentimage data, and the number of bits which is the number of colors thateach dot has are written. Specifically, “2020×1624 dots” is written asthe number of dots, and “24 bits” is written as the number of bits.

In the category of photographing information, a date and time ofphotographing, exposure time, F number, ISO speed, focal distance andthe like are written. Here, “2014/11/28 14:22:22” is written as the dateand time of photographing, “1/500” is written as the exposure time, “4”is written as the F number, “200” is written as the ISO speed, and “50mm” is written as the focal distance.

In the category of image-capturing system color space information, alight source characteristic, peak wavelength λa, peak wavelength λb,peak wavelength λc, primary stimulus value 1 (vertex AB), primarystimulus value 2 (vertex CD), primary stimulus value 3 (vertex EF),reference point, brightness correction, color processing, display colorspace, lower limit value and upper limit value of a target band, and thelike are written.

The light source characteristic indicates the type of light source in aphotographing condition. Here, “halogen light” is written. The peakwavelength λa is the peak wavelength of the NIR1 filter. Here, “1150 nm”is written. The peak wavelength λb is the peak wavelength of the NIR2filter. Here, “1400 nm” is written. The peak wavelength λc is the peakwavelength of the NIR3 filter. Here, “1650 nm” is written. The region302 explained with reference to FIG. 4 can be defined by these peakwavelengths.

The primary stimulus value 1 is determined by the peak wavelength λa andthe half width of the NIR1 filter, and the photosensitivity of the imagesensor 100. Likewise, the primary stimulus value 2 is determined by thepeak wavelength λb and the half width of the NIR2 filter, and thephotosensitivity of the image sensor 100. The primary stimulus value 3is determined by the peak wavelength λc and the half width of the NIR3filter, and the photosensitivity of the image sensor 100. Although asexplained already with reference to FIG. 4, the primary stimulus value 1(vertex AB) is positioned on a straight line linking the peak wavelengthλa and the reference point RefP, it does not have to be at a position ona straight line linking the peak wavelength λa and the reference pointRefP. The same applies to the primary stimulus value 2 (vertex CD) andthe primary stimulus value 3 (vertex EF). For example, it is alsopossible to set any of or all the primary stimulus values 1 to 3 to anyvalues according to an object, an illumination environment or the like.Because as explained with reference to FIG. 4 and the like already, avirtual color coordinate system is defined along the coordinate axes ofx′, y′, the primary stimulus value 1, the primary stimulus value 2 andthe primary stimulus value 3 can be represented by coordinates. In moredetail, the primary stimulus value 1 (vertex AB), the primary stimulusvalue 2 (vertex CD) and the primary stimulus value 3 (vertex EF)correspond to the respective coordinates of the vertexes of the region303, explained with reference to FIG. 4. Here, (0.17, 0.20) is writtenas the primary stimulus value 1 (vertex AB), (0.13, 0.65) is written asthe primary stimulus value 2 (vertex CD), and (0.63, 0.35) is written asthe primary stimulus value 3 (vertex EF). By associating the primarystimulus value 1 (vertex AB), the primary stimulus value 2 (vertex CD)and the primary stimulus value 3 (vertex EF) with the correspondingvertexes among the vertex GH, the vertex IJ and the vertex KL,respectively, in the visible color space explained with reference toFIG. 5, for example, the virtual color space and the visible color spacecan be associated with each other.

In the reference point, “(0.33, 0.33)” is written as the coordinate ofthe reference point. This coordinate is a coordinate in the case wherethe reference point is set to the default, and is a coordinatecorresponding to the reference white. If the reference point is set by auser, the coordinate of the set reference point is written. “γ=1” iswritten in the brightness correction.

The color processing indicates the resolution of display. In the colorprocessing, “separate water and oil” is written. From this information,the gain corresponding to objects to be targets of color-discriminationcan be determined, and the resolution of display can be determined.

The display color space indicates the color gamut that is to be set in adisplay system. Here, “sRGB” is written in the display color space. Thisinformation include the coordinate of the reference white RefW, and thecoordinates of the three primary stimulus values in sRGB. In the exampleof the visible color space explained with reference to FIG. 5, (0.33,0.33) is written as the coordinate (o, p) of the reference white RefW.Also, here, (0.14, 0.06) is written as the coordinate (g, h) of thevertex GH, (0.20, 0.68) is written as the coordinate (i, j) of thevertex IJ and (0.63, 0.32) is written as the coordinate (k, l) of thevertex KL. If information indicative of another color gamut such asAdobe RGB is written in the display color space, different values fromthose mentioned above are written as the three primary stimulus values.As described in detail below, based on the information in the displaycolor space, the virtual color space three primary stimulus values andthe visible color space three primary stimulus values can be associatedwith each other, and the reference point and the reference white can beassociated with each other.

The lower limit value and upper limit value of the target band indicatethe range of the output characteristic of the image sensor 100. Here,“850 nm” is written as the lower limit value, and “1950 nm” is writtenas the upper limit value. As mentioned above, the region 301 explainedwith reference to FIG. 4 is defined by this information.

If the infrared wavelength information like the one described above isgenerated, the infrared wavelength information can be converted into agenerally used image format according to a rule of conversion into thegenerally used image format as long as such a conversion rule isdetermined. That is, as explained already, if the virtual color space isassociated with the visible color space, the same conversion can beperformed for the infrared wavelength information related to aparticular object. Thereby, a particular visible color is associatedwith a particular object. Because the same object is to be displayed inthe same color, the reproducibility of reproduction can be ensured. Forexample, water as an object is displayed in bluish colors, and oil canbe displayed in whitish colors. Also, regardless of the type of displaydevice, the compatibility of reproduction can be ensured. From what isdescribed above, infrared wavelength information can also be generatedas information necessary for conversion into a generally used imageformat.

Although in the explanation above, the image sensor 100 includes threetypes of bandpass filter with which the lower limit value and upperlimit value, as its output characteristic, become 850 nm and 1950 nm,respectively, it may include other bandpass filters. Hereinafter, thedetails are explained.

FIG. 9 is a figure for explaining a relationship between the type ofbandpass filter and an image-capturing system color space. (a) of FIG. 9is a figure showing a sensitivity distribution of the outputcharacteristic of the image sensor 100 already explained. The horizontalaxis indicates wavelength [nm], and the vertical axis indicatesphotosensitivity. The lower limit value of the output characteristic is850 nm, and its upper limit value is 1950 nm. (b) of FIG. 9 is a figureobtained by extracting the region 301, the region 302 and the region 303in the figure shown in FIG. 4 for simplification of the figure. (c) ofFIG. 9 is a figure showing a sensitivity distribution of the outputcharacteristic of an image sensor provided with three other types ofbandpass filter. The half widths of the three types of bandpass filterprovided to the image sensor are narrower than the half widths of thethree types of bandpass filter provided to the image sensor 100. Thehorizontal axis indicates wavelength [nm], and the vertical axisindicates photosensitivity. The lower limit value of the outputcharacteristic is 1000 nm, and its upper limit value is 1800. (d) ofFIG. 9 is a figure showing a region 601, a region 602 and a region 603corresponding to the region 301, the region 302 and the region 303according to the output characteristic shown in (c) of FIG. 9.

In the visible color space already explained, the purities of colorsincrease as the distance from the reference white RefW increases. Inother words, a color of higher purity can be expressed as a visiblecolor space region becomes larger. The same also applies to a case ofgenerating an infrared wavelength band-based image. That is, a visiblecolor space region becomes larger as a corresponding virtual color spaceregion becomes larger; as a result, a color of higher purity can beexpressed.

It can be known from comparison between the sensitivity distribution in(a) of FIG. 9 and the sensitivity distribution in (c) of FIG. 9 that thehalf width of each sensitivity distribution shown in the sensitivitydistribution of (c) of FIG. 9 is narrower than the half width of eachsensitivity distribution shown in (a) of FIG. 9. Accordingly, each pixelof an image sensor having the sensitivity distribution of (c) of FIG. 9as its output characteristic has a pixel value based on the wavelengthof a more restricted bandwidth. As a result, a color of higher puritycan be expressed. Accordingly, the area of the virtual color spaceregion 603 in (d) of FIG. 9, that is, the color-saturation becomeslarger than the area of the virtual color space region 303 in (b) ofFIG. 9, that is, the color-saturation.

Although in the present example, the lower limit value and upper limitvalue of the output characteristic shown in (a) of FIG. 9 are mutuallydifferent from the lower limit value and upper limit value of the outputcharacteristic shown in (c) of FIG. 9, the shape of the region 301 andthe shape of the region 601 are the same. That is, the lower limit valueand upper limit value of the output characteristic are variable, and theregion 301 and the region 601 have different pitch widths betweenadjacent wavelengths.

Although in the explanation above, the image sensor 100 is configured toinclude three types of bandpass filter, it may be configured to includefurther different three types of bandpass filter. That is, two types offilter set may be provided in a mixed manner. In this case, a filter setthat allows easy color-discrimination may be selected according to anobject. By generating a difference in hue to a certain extent at thestage of spectral diffraction, a more effective process can be expectedat the stage of adjusting the position of a reference point so as toallow easy color-discrimination.

FIG. 10 is a figure for explaining a relationship between bandpassfilters and object spectra. In the present example, the image sensor 100is photosensitive to light in the range of 700 nm to 2000 nm. (a) ofFIG. 10 is a figure showing the sensitivity distribution of the outputcharacteristic of the image sensor 100 already explained and the spectraof an object P and an object Q. The horizontal axis indicates wavelength[nm], and the vertical axis indicates photosensitivity. The lower limitvalue of the output characteristic is 850 nm, and its upper limit valueis 1950 nm. (b) of FIG. 10 is a figure obtained by extracting the region301, the region 302 and the region 303 in the figure shown in FIG. 4 forsimplification of the figure. (c) of FIG. 10 is a figure showing thesensitivity distribution of the output characteristic in the case wheredifferent three types of bandpass filter are selected and the spectra ofthe object P and the object Q. The waveform of the sensitivitydistribution shown in (c) of FIG. 10 and the waveform of the sensitivitydistribution shown in (a) of FIG. 10 are approximately the same.However, as compared with the bandpass filters shown in FIG. 3, thedifferent three types of bandpass filter allow passage of a wavelengthband on a shorter wavelength side. The horizontal axis indicateswavelength [nm], and the vertical axis indicates photosensitivity. Thelower limit value of the output characteristic is 700 nm, and its upperlimit value is 1800 nm. The peak wavelength λd is 1000 nm, the peakwavelength is λe is 1250, and the peak wavelength λf is 1500. (d) ofFIG. 10 is a figure showing a region 701, a region 702 and a region 703corresponding to the region 301, the region 302 and the region 303corresponding to the output characteristic shown in (c) of FIG. 10.Also, in (b) of FIG. 10 and (d) of FIG. 10, a region p indicates aregion in which the object P is mapped to the virtual color space, and aregion q indicates a region in which the object Q is mapped to thevirtual color space.

Depending on how the filters are set in relation to the spectra of theobject P and the object Q, the regions in which the object P and theobject Q are mapped to the virtual color space change. Comparisonbetween (a) of FIG. 10 and (c) of FIG. 10 shows that a gap t between theobject P and the object Q is wider than a gap r between the object P andthe object Q. Likewise, a gap u between the object P and the object Q iswider than a gap s between the object P and the object Q.

As shown in (c) of FIG. 10, if a filter is set to a portion where a gapbetween the object P and the object Q is relatively small, as shown in(d) of FIG. 10, the region p and the region q are mapped to regionsclose to each other. Because a large difference is not generated in hue,it may be difficult to perform color-discrimination in some cases.

On the other hand, as shown in (a) of FIG. 10, if a filter is set to aportion where a gap between the object P and the object Q is relativelylarge, as shown in (b) of FIG. 10, the region p and the region q aremapped to regions distant from each other. Because a large difference isgenerated in hue, it becomes easy to perform color-discrimination.

FIG. 11 is a flow diagram showing a process flow of the digital camera10. The present flow is started when a user turns on a power source.

The control unit 201 judges whether or not the SW1 is pressed (StepS101). If it is judged that the SW1 is pressed (YES at Step S101), thecontrol unit 201 judges whether or not the SW2 is pressed (Step S102).If it is judged that the SW2 is pressed (YES at Step S102), the controlunit 201 performs a process of photographing (Step S103).

The control unit 201 reads out, as image-capturing parameters from thesystem memory 206, information indicative of the transmittances of threetypes of bandpass filter and information indicative of thephotosensitivity of the image sensor 100. Then, it calculates the lowerlimit value and upper limit value of the output characteristic (StepS104). For example, as explained in association with FIG. 3, the outputcharacteristic is calculated by multiplying the transmittances of thebandpass filters by the photosensitivity of the image sensor 100.Thereby, the region 301 shown in FIG. 4 is defined.

The control unit 201 judges whether there is an instruction about areference point from a user (Step S105). If it is judged that there isan instruction about a reference point (YES at Step S105), a referencepoint is set following the instruction from the user (Step S106). Forexample, as explained in association with FIG. 7, a reference point isset within the virtual color space. If it is judged that there is not aninstruction about a reference point (NO at Step S105), a reference pointis set at a default position (Step S107). For example, as explained inassociation with FIG. 5, a reference point is set at a positioncorresponding to the reference white.

The image processing unit 205 generates main image-capturing data byperforming various types of processing on the image-capturing dataobtained by digital conversion (Step S108). On the other hand, thecontrol unit 201 generates infrared wavelength information (Step S109).For example, it generates the infrared wavelength information followingthe format shown in FIG. 8.

The image processing unit 205 relates the infrared wavelengthinformation to the main image-capturing data (Step S110) to make theminto a file; thereby, it generates an image-capturing file (Step S111).The image processing unit 205 stores the image-capturing file in thememory card 220 (Step S112). The control unit 201 converts the infraredwavelength information into visible color space information (Step S113).For example, as explained with reference to FIG. 5, a virtual colorspace is converted into a visible color space, a reference point isconverted into a reference white, and so on. The image processing unit205 performs image processing on the main image-capturing data byreferring to the visible color space information as explained already(Step S114) to convert data obtained after the image processing intodisplay signals by the LCD drive circuit 210, and thereafter displays iton the display unit 209 (Step S115). The control unit 201 judges whetherthe power source is turned off (Step 116). If it is judged that thepower source remains turned on (NO at Step S116), it proceeds to StepS101, and if it is judged that the power source is turned off (YES atStep S116), the series of processing ends.

Although in the explanation above, the control unit 201 relates infraredwavelength information as tag information to image-capturing data, itmay relate the information as a link file to the image-capturing data.Although in the explanation above, the CIE color coordinate system wasexplained as an example of a visible color coordinate system, it may beanother color coordinate system.

Although in the explanation above, the digital camera 10 as one exampleof an image-capturing device is configured to include the display unit209, the image-capturing device may be configured not to include thedisplay unit 209. In this case, the image-capturing device may transmitan image-capturing file to another device including a display unit. Theother device can convert infrared wavelength-based image-capturing datainto image data and display it by performing the processes from StepS113 to Step S115 in FIG. 11.

Although in the explanation above, bandpass filters are provided to theimage sensor 100, the location at which bandpass filters are arranged isnot limited to the image sensor 100. Bandpass filters may be provided asa filter unit crossing the optical axis 21 at a later stage of thetaking lens 20. In this case, following user setting, the control unit201 arranges three types of bandpass filter in order in the range of anobject light flux and performs a photographing operation insynchronization with the arrangement of the respective ones. Then, theimage processing unit 205 acquires sequentially three pieces ofimage-capturing data as image-capturing plain data configured withimage-capturing data of all the pixels from the image sensor 100.Because with a configuration using a filter unit, image-capturing datacorresponding to each one among the three types of bandpass filter canbe obtained for all the pixels, the above-mentioned interpolationprocess does not have to be performed.

If the digital camera 10 is a digital camera whose lens is replaceable,and filter units are configured integral with replaceable lenses,image-capturing parameters may be stored in a lens memory in thereplaceable lenses. The camera body may acquire image-capturingparameters from the lens memories.

Although in the explanation above, a reference point is associated witha reference white, because the reference point is a point just set forthe purpose of separating colors, it does not necessarily have to beassociated with the reference white. For example, if two objects are tobe color-discriminated, it may be set right in the middle of regions ofthe two objects mapped to the virtual color space. Also, although thereference point is set in the region 303, it may be set outside theregion 303 as long as it is set in the region 302. Although in theexplanation above, the coordinate of a reference point is variable byuser setting, it may be fixed. In this case, the items of referencevalues do not have to be present in the format explained with referenceto FIG. 8. Also, from the perspective of to which visible wavelengthband each pixel value of image-capturing data is to be related, theitems about the vertex AB, the vertex CD and the vertex EF do not haveto be present. Although in the explanation above, the positions of thepeak wavelength λa, the peak wavelength λb and the peak wavelength λcare determined uniquely by setting three points, which are the lowerlimit value, the upper limit value and the reference point of a targetregion, three points which are the lower limit value, the upper limitvalue and a point other than the reference point may be set. Forexample, by setting the coordinate of the peak wavelength λb, theremaining peak wavelength λa and peak wavelength λc may be determineduniquely.

Although in the explanation above, the virtual color space is definedwith x′ as the horizontal axis and y′ as the vertical axis, the virtualcolor space is first of all obtained by defining the infrared wavelengthregion space virtually as a color space, it may be defined with otheraxes as the horizontal axis and the vertical axis, respectively.Although in the explanation above, pitch widths of wavelengths from thelower limit value to the upper limit value of the region 301 are set ina manner similar to that for the region 401, they may be set to equalwidths. Although in the explanation above, the lower limit value andupper limit value of the region 301 are variable, the lower limit valueand upper limit value may be fixed. In this case, the shape of theregion 301 becomes variable.

Although in the explanation above, in order to associate the virtualcolor space and the visible color space with each other, a referencepoint and a reference white are associated with each other after thelower limit values of target bands are associated with each other andupper limit values of the target bands are associated with each other,respectively, virtual color space three primary stimulus values andvisible color space three primary stimulus values may be associated witheach other, and additionally a reference point and a reference white maybe associated with each other. In this case, the lower limit values ofthe target bands do not necessarily have to be associated with eachother, and the upper limit values of the target bands do not necessarilyhave to be associated with each other. This is because thehorseshoe-shape region 401 is just a range determined by monochromaticlight, and the visible color space is represented by the region 403.Also, if virtual color space three primary stimulus values and visiblecolor space three primary stimulus values are associated with eachother, information indicative of the visible color space primarystimulus value to which each one among the virtual color space threeprimary stimulus values is associated may be written in theimage-capturing system color space information.

FIG. 12 is a figure showing one example of image-capturing system colorspace information. In the category of image-capturing system color spaceinformation in (a) of FIG. 12, in addition to the light sourcecharacteristic, peak wavelength λa, peak wavelength λb, peak wavelengthλc, primary stimulus value 1 (vertex AB), primary stimulus value 2(vertex CD), primary stimulus value 3 (vertex EF), reference point,brightness correction, color processing, display color space, and lowerlimit value and upper limit value of a target band already explainedwith reference to FIG. 8, the corresponding primary stimulus value 1(vertex GH), the corresponding primary stimulus value 2 (vertex IJ) andthe corresponding primary stimulus value 3 (vertex KL) are written. Thevirtual color space vertexes (that is, the vertex AB, the vertex CD andthe vertex EF), the visible color space vertexes (that is, the vertexGH, the vertex IJ and the vertex KL), the reference point RefP and thereference white RefW are the same as those explained with reference toFIG. 5.

The corresponding primary stimulus value 1 (vertex GH) indicates thevisible color space primary stimulus value with which the virtual colorspace primary stimulus value 1 is associated. Likewise, thecorresponding primary stimulus value 2 indicates the visible color spaceprimary stimulus value with which the virtual color space primarystimulus value 2 is associated, and the corresponding primary stimulusvalue 3 indicates the visible color space primary stimulus value withwhich the virtual color space primary stimulus value 3 is associated.Here, the coordinate “(0.14, 0.06)” of the vertex GH is written as thecorresponding primary stimulus value 1, the coordinate “(0.20, 0.68)” ofthe vertex IJ is written as the corresponding primary stimulus value 2and the coordinate “(0.63, 0.32)” of the vertex KL is written as thecorresponding primary stimulus value 3. That is, the virtual color spaceprimary stimulus value 1 (vertex AB) is associated with the visiblecolor space vertex GH, the virtual color space primary stimulus value 2(vertex CD) is associated with the visible color space vertex IJ and thevirtual color space primary stimulus value 3 (vertex EF) is associatedwith the visible color space vertex KL. Also, the coordinate of thereference point RefP is associated with (0.33, 0.33). As explainedalready, (0.33, 0.33) is the coordinate of the reference point RefP ifit is set to a default, and a user can freely set the coordinate of thereference point. In this case also, the coordinate of the referencepoint RefP is associated with (0.33, 0.33) which is the coordinate ofthe reference white RefW.

Also, as explained with reference to FIG. 8 already, the display colorspace includes the coordinate of the reference white RefW. Accordingly,based on this information, it is possible to associate the coordinate ofthe reference point RefP with the coordinate of the reference whiteRefW. As described below, there are more than one combinations ofvirtual color space three primary stimulus values and visible colorspace three primary stimulus values that are related to each other. Inview of this, in the example, in order to uniquely determine acorrespondence, information indicative of visible color space primarystimulus values with which they are associated is written. In a similarmanner to that for associating the virtual color space and the visiblecolor space with each other, if coordinates representing an object aremapped to the region 303, the coordinates can be converted intocoordinates in the region 403. That is, they can be converted intovisible colors.

Although in the explanation above, the respective virtual color spacethree primary stimulus values are associated with the correspondingvisible color space primary stimulus values with the correspondingstimulus value 1, the corresponding stimulus value 2 and thecorresponding stimulus value 3, the virtual color space three primarystimulus values and the visible color space three primary stimulusvalues may be related to each other by another method. For example, thevirtual color space three primary stimulus values and the visible colorspace three primary stimulus values can be associated with each other byutilizing the order in which the three primary stimulus values arewritten. Specifically, a primary stimulus value written first isassociated with the coordinate (0.14, 0.06) of the vertex GH in thevisible color space, a primary stimulus value written second isassociated with the coordinate (0.20, 0.68) of the vertex IJ in thevisible color space, and a primary stimulus value written third isassociated with the coordinate (0.63, 0.32) of the vertex KL in thevisible color space.

Here, as explained already, if “sRGB” is written as the display colorspace, (0.14, 0.06) is written as the coordinate (g, h) of the vertexGH, (0.20, 0.68) is written as the coordinate (i, j) of the vertex IJ,and (0.63, 0.32) is written as the coordinate (k, l) of the vertex KL;on the other hand, if information indicative of another color gamut iswritten, different values from those described above are written asthree primary stimulus values. That is, information written as thedisplay color space and visible color space three primary stimulusvalues correspond to each other in a one-to-one relationship.Accordingly, if the order in which three primary stimulus values arewritten is utilized, only a display color space has to be designatedeven if visible color space three primary stimulus values are notwritten for each piece of image data. The display system can associatevirtual color space three primary stimulus values with visible colorspace three primary stimulus values according to a designated displaycolor space following a rule related to the above-mentioned order inwhich three primary stimulus values are written.

Using (a) of FIG. 12 explained already as an example, the primarystimulus value 1 (vertex AB), the primary stimulus value 2 (vertex CD)and the primary stimulus value 3 (vertex EF) are written in this order.Accordingly, even if the corresponding stimulus value 1 is not written,the primary stimulus value 1 (vertex AB) written first is associatedwith the coordinate (0.14, 0.06) of the vertex GH in the visible colorspace. Likewise, even if the corresponding stimulus value 2 is notwritten, the primary stimulus value 2 (vertex CD) written second isassociated with the coordinate (0.20, 0.68) of the vertex IJ in thevisible color space, and even if the corresponding stimulus value 3 isnot written, the primary stimulus value 3 (vertex EF) written third isassociated with the coordinate (0.63, 0.32) of the vertex KL in thevisible color space. As described above, if virtual color space threeprimary stimulus values and the visible color space three primarystimulus values are associated with each other by utilizing the order inwhich the three primary stimulus values are written, the image-capturingsystem color space information in (a) of FIG. 12 does not have toinclude the corresponding stimulus value 1, the corresponding stimulusvalue 2 and the corresponding stimulus value 3.

In the explanation above, the virtual color space three primary stimulusvalues and the visible color space three primary stimulus values arerespectively associated according to wavelengths. That is, primarystimulus values that are on the short wavelength side are related toeach other, primary stimulus values on the long wavelength side arerelated to each other, and primary stimulus values of wavelengthsbetween the short wavelength side and the long wavelength side arerelated to each other, but a manner of relating them is not limited tothis. Hereinafter, the details are explained. In the following example,a case where the virtual color space three primary stimulus values andthe visible color space three primary stimulus values are associatedwith each other by utilizing the order in which the three primarystimulus values are written is explained. Accordingly, as illustrated inthe figure, the image-capturing system color space information in (b) ofFIG. 12 does not include the corresponding stimulus value 1, thecorresponding stimulus value 2 and the corresponding stimulus value 3.The same also applies to a case where the virtual color space threeprimary stimulus values and the visible color space three primarystimulus values are associated with each other using the correspondingstimulus value 1, the corresponding stimulus value 2 and thecorresponding stimulus value 3.

The order in which the three primary stimulus values are written in theimage-capturing system color space information in (b) of FIG. 12 isdifferent from the order in which the three primary stimulus values arewritten in the image-capturing system color space information in (a) ofFIG. 12. Specifically, they are written in the order of the primarystimulus value 2 (vertex CD), the primary stimulus value 1 (vertex AB)and the primary stimulus value 3 (vertex EF). Accordingly, the primarystimulus value 2 (vertex CD) written first is associated with thecoordinate (0.14, 0.06) of the vertex GH in the visible color space.Likewise, the primary stimulus value 1 (vertex AB) written second isassociated with the coordinate (0.20, 0.68) of the vertex IJ in thevisible color space, and the primary stimulus value 3 (vertex EF)written third is associated with the coordinate (0.63, 0.32) of thevertex KL in the visible color space. By relating them in the mannerdescribed above, colors allocated to the virtual color space threeprimary stimulus values can be changed. Specifically, because thecoordinate (0.13, 0.65) of the primary stimulus value 2 (vertex CD) isassociated with the coordinate (0.14, 0.06) of the vertex GH, it isassociated with a bluish color. Because the coordinate (0.17, 0.20) ofthe primary stimulus value 1 (vertex AB) is associated with thecoordinate (0.20, 0.68) of the vertex IJ, it is associated with agreenish color. Because the coordinate (0.63, 0.35) of the primarystimulus value 3 (vertex EF) is associated with the coordinate (0.63,0.32) of the vertex KL, it is associated with a reddish color.

Although in the explanation above, the near infrared band is mentionedas an example of the invisible band, it may be another band as along asit is a band outside the visible band. For example, if the image sensor100 is photosensitive to light in the ultraviolet band, the control unit201 can generate ultraviolet wavelength information for convertingultraviolet band-based image-capturing data into visible color spaceimage data.

FIG. 13 is a figure for explaining a correspondence between animage-capturing system color space and a display system color space.

(a) of FIG. 13 is a figure for explaining a space defined by the outputcharacteristic of the image sensor 100. Similar to FIG. 4, itillustrates x′, y′ corresponding to a chromaticity coordinate, x, y ascoordinate axes. The horizontal axis is x′, and the vertical axis is y′.In the present example, as one example, the lower limit value of theoutput characteristic is 200 nm and its upper limit value is 400 nm.That is, the space defined by the output characteristic is anultraviolet band space. (b) of FIG. 13 shows a visible color coordinatesystem chromaticity diagram. The horizontal axis is x, and the verticalaxis is y. In the present example, a case where the virtual color spacethree primary stimulus values and the visible color space three primarystimulus values are associated with each other, and the reference pointand the reference white are associated with each other is explained.

The vertex AB which is the virtual color space primary stimulus value 1is associated with the visible color space vertex GH. Likewise, thevertex CD which is the virtual color space primary stimulus value 2 isassociated with the visible color space vertex IJ, and the vertex EFwhich is the virtual color space primary stimulus value 3 is associatedwith the visible color space vertex KL. Also, the reference point RefPis associated with the reference white RefW. Similar to the case of theinfrared wavelength band, if coordinates representing an object aremapped to the region 303, the coordinates can be converted intocoordinates in the region 403. That is, they can be converted intovisible colors.

Although in the explanation above, the NIR1 filter and the NIR3 filterare transmissive to the peak wavelength of the NIR2 filter, and the NIR2filter is transmissive to the respective peak wavelengths of the NIR1filter and the NIR3 filter, in the output characteristic of the imagesensor 100 explained already with reference to (c) of FIG. 3 (forexample, the range within which it has a photosensitivity equal to orhigher than 1%), all of the NIR1 filter, the NIR2 filter and the NIR3filter preferably have transmittances equal to or higher than 1%. Withsuch a configuration, a pixel provided with the NIR1 filter has a highsensitivity around 1150 nm, and has a certain degree of sensitivityaround the upper limit value. Likewise, a pixel provided with the NIR2filter has a high sensitivity around 1400 nm, and has certain degree ofsensitivity around the lower limit value and upper limit value, and apixel provided with the NIR3 filter has a high sensitivity around 1650nm, and has a certain degree of sensitivity around the lower limitvalue.

If the image processing unit 205 converts infrared wavelength band-basedimage-capturing data into visible wavelength band-based image-capturingdata to generate the image data, not only the NIR1 filter, the NIR2filter and the NIR3 filter are associated with mutually differentvisible wavelength bands, respectively, but also respective wavelengthband pixel signals are used and converted into visible wavelength bandpixel signals. For example, a pixel signal of the wavelength band of theNIR1 filter is converted into a pixel signal of a visible wavelengthband using, in addition to the pixel signal, a pixel signal of thewavelength band of the NIR2 filter and a pixel signal of the wavelengthband of the NIR3 filter. In this manner, by using pixel signals of allthe wavelength bands of the NIR1 filter, the NIR2 filter and the NIR3filter, color representation at a high resolution is made possible.

In order to enhance the resolution of color representation, all of theNIR1 filter, the NIR2 filter and the NIR3 filter preferably havetransmittances equal to or higher than 1% in the output characteristicof the image sensor 100 as mentioned above. However, this is not theonly configuration that enhances the resolution of color representation.At least, in one possible configuration, at the peak wavelength of thewavelength band of the NIR1 filter which is on the shortest wavelengthside, the wavelength band of the NIR3 filter which is on the longestwavelength side overlaps the former wavelength band. Specifically, at1150 nm which is the peak wavelength of the wavelength band of the NIR1filter, the NIR3 filter may have a transmittance equal to or higher than1%. On the other hand, in one possible configuration, at least, at thepeak wavelength of the wavelength band of the NIR3 filter which is onthe longest wavelength side, the wavelength band of the NIR1 filterwhich is on the shortest wavelength side overlaps the former wavelengthband. Specifically, at 1650 nm which is the peak wavelength of thewavelength band of the NIR3 filter, the NIR1 filter may have atransmittance equal to or higher than 1%. With such a combination also,because the NIR1 filter, the NIR2 filter and the NIR3 filter overlap oneanother over wavelength bands at least in a range including therespective peak wavelengths, color representation in multiple colorsbecomes possible.

FIG. 14 is a figure for explaining one example of color allocation. (a)of FIG. 14 is the same as (a) of FIG. 5, and (b) of FIG. 14 is the sameas (b) of FIG. 5.

In the example of FIG. 14, the coordinate (a, b) of the point AB isassociated with the coordinate (g, h) of the point GH. That is,coordinates around the coordinate (a, b) of the point AB are associatedwith bluish colors. The coordinate (c, d) of the point CD is associatedwith the coordinate (i, j) of the point IJ. That is, coordinates aroundthe coordinate (c, d) of the point CD are associated with greenishcolors. The coordinate (e, f) of the point EF is associated with thecoordinate (k, l) of the point KL. That is, coordinates around thecoordinate (e, f) of the point EF are associated with reddish colors.

As explained already using (a) of FIG. 7, because the spectral intensityof water increases rapidly in the band up to around 1100 nm, anddecreases rapidly in the band after around 1250 nm, it differssignificantly from the spectral intensity of oil in the band up toaround 1100 nm and the band after around 1200 nm. Because the spectralintensity corresponding to the wavelength bands of the NIR2 filter andthe NIR3 filter is relatively low, water Objw is mapped to a positionclose to the point GH in the color coordinate system chromaticitydiagram. Thus, it is recognized in bluish colors overall in an imageafter conversion into the visible wavelength band. On the other hand,because the spectral intensity of oil ranges over a relatively widerange in the near infrared wavelength band, oil Objo is mapped to aposition close to the reference white RefW in the color coordinatesystem chromaticity diagram. Thus, it is recognized in somewhat whitishcolor as compared with water in an image after conversion into thevisible wavelength band.

FIG. 15 is a figure for explaining another example of color allocation.(a) of FIG. 15 is the same as (a) of FIG. 5, and (b) of FIG. 15 is thesame as (b) of FIG. 5.

In the example of FIG. 15, the coordinate (a, b) of the point AB isassociated with the coordinate (i, j) of the point IJ. That is,coordinates around the coordinate (a, b) of the point AB are associatedwith greenish colors. The coordinate (c, d) of the point CD isassociated with the coordinate (k, l) of the point KL. That is,coordinates around the coordinate (c, d) of the point CD are associatedwith reddish colors. The coordinate (e, f) of the point EF is associatedwith the coordinate (g, h) of the point GH. That is, coordinates aroundthe coordinate (e, f) of the point EF are associated with bluish colors.

In this example, the water Objw is mapped to a position close to thepoint IJ in the color coordinate system chromaticity diagram. Thus, itis recognized in greenish colors overall in an image after conversioninto the visible wavelength band. On the other hand, the oil Objo ismapped to a position close to the reference white RefW in the colorcoordinate system chromaticity diagram. Thus, it is recognized insomewhat whitish color as compared with water in an image afterconversion into the visible wavelength band.

FIG. 16 is a figure for explaining another example of color allocation.(a) of FIG. 16 is the same as (a) of FIG. 5, and (b) of FIG. 16 is thesame as (b) of FIG. 5.

In the example of FIG. 16, the coordinate (a, b) of the point AB isassociated with the coordinate (k, l) of the point KL. That is,coordinates around the coordinate (a, b) of the point AB are associatedwith reddish colors. The coordinate (c, d) of the point CD is associatedwith the coordinate (g, h) of the point GH. That is, coordinates aroundthe coordinate (c, d) of the point CD are associated with bluish colors.The coordinate (e, f) of the point EF is associated with the coordinate(i, j) of the point IJ. That is, coordinates around the coordinate (e,f) of the point EF are associated with greenish colors.

In this example, the water Objw is mapped to a position close to thepoint KL in the color coordinate system chromaticity diagram. Thus, itis recognized in reddish colors overall in an image after conversioninto the visible wavelength band. On the other hand, the oil Objo ismapped to a position close to the reference white RefW in the colorcoordinate system chromaticity diagram. Thus, it is recognized insomewhat whitish color as compared with water in an image afterconversion into the visible wavelength band.

As explained above, depending on how the virtual color space threeprimary stimulus values and the visible color space three primarystimulus values are related to each other, an observation target can berepresented in different colors. Humans are known to be generally moresensitive to differences in bluish colors and reddish colors than togreenish colors. In view of this, in the present embodiment, theobservation information for color-discriminating water and oil indicatesthat the coordinate (a, b) of the point AB is associated with thecoordinate (k, l) of the point KL, the coordinate (c, d) of the point CDis associated with the coordinate (g, h) of the point GH and thecoordinate (e, f) of the point EF is associated with the coordinate (i,j) of the point IJ. By performing color allocation according to thisobservation information, the image processing unit 205 can allocate aprimary stimulus value corresponding to the highest spectral intensityamong spectral intensities corresponding to the respective wavelengthbands of the NIR1 filter, the NIR2 filter and the NIR3 filter to thepoint KL, that is, a reddish color. The observation information may begenerated to indicate that the coordinate (a, b) of the point AB isassociated with the coordinate (g, h) of the point GH, that is, a bluishcolor, the coordinate (c, d) of the point CD is associated with thecoordinate (i, j) of the point IJ, and the coordinate (e, f) of thepoint EF is associated with the coordinate (k, l) of the point KL. Asdescribed above, by switching over manners of relating colors inaccordance with the human visual characteristics corresponding to anobject spectrum as appropriate, it becomes possible to makecolor-discrimination of an object easy in a visible image.

FIG. 17 is a flow diagram showing a process flow of the digital camera10. The present flow is started when a user turns on a power source. Itis assumed that water and oil are set as two observation targets to betargets of color-discrimination through a menu screen related to settingof observation targets explained already, and a user photographs waterand oil.

The control unit 201 judges whether or not the SW1 is pressed (StepS201). If it is judged that the SW1 is pressed (YES at Step S201), thecontrol unit 201 judges whether or not the SW2 is pressed (Step S202).If it is judged that the SW2 is pressed (YES at Step S202), the controlunit 201 performs a process of photographing (Step S103).

The image processing unit 205 generates main image-capturing data byperforming various types of processing on the image-capturing dataobtained by digital conversion (Step S204). The image processing unit205 acquires, from the system memory 206, observation informationcorresponding to a set observation target (Step S205). Here, observationinformation for color-discriminating water and oil is acquired. Theimage processing unit 205 acquires virtual color space three primarystimulus values and visible color space three primary stimulus values(Step S206). The image processing unit 205 uses the acquired observationinformation, and the virtual color space three primary stimulus valuesand the visible color space three primary stimulus values to determinecolor allocation (Step S207). As explained already with reference toFIGS. 14 to 16, for example, the coordinate (a, b) of the point AB isassociated with the coordinate (k, l) of the point KL, the coordinate(c, d) of the point CD is associated with the coordinate (g, h) of thepoint GH and the coordinate (e, f) of the point EF is associated withthe coordinate (i, j) of the point IJ. Then, image processing isperformed on the main image-capturing data according to the determinedcolor allocation to generate color image data (Step S208).

After converting the color image data into display signals by the LCDdrive circuit 210, it is displayed on the display unit 209 (Step S209).The control unit 201 judges whether the power source is turned off (StepS210). If it is judged that the power source remains turned on (NO atStep S210), it proceeds to Step S201, and if it is judged that the powersource is turned off (YES at Step S210), the series of processing ends.

Although in the explanation above, the image processing unit 205associates the respective virtual color space three primary stimulusvalues with points in the visible color space, at least one of thevirtual color space three primary stimulus values may be associated witha point outside the visible color space. FIG. 18 is a figure forexplaining another example of a correspondence between animage-capturing system color space and a display system color space. (a)of FIG. 18 is the same as (a) of FIG. 5. (b) of FIG. 18 shows a visiblecolor coordinate system chromaticity diagram. The horizontal axis is x,and the vertical axis is y.

In the example of FIG. 18, the coordinate (a, b) of the point AB isassociated with the coordinate (g, h) of the point GH. The coordinate(c, d) of the point CD is associated with the coordinate (i′, j′) of thepoint I′J′. The point I′J′ is a point outside the region 403 indicatingthe visible color space. The coordinate (e, f) of the point EF isassociated with the coordinate (k, l) of the point KL. In the region401, the color-saturation becomes higher as the distance to the outsidebecomes shorter. Accordingly, by associating the coordinate (c, d) ofthe point CD with the coordinate (i′, j′) of the point I′J′ locatedoutward from the coordinate (i, j) of the point IJ, the greenishcolor-saturation can be improved. In more detail, if the coordinate (c,d) of the point CD is associated with the coordinate (i, j) of the pointIJ, an observation target is expressed with whitish colors overall ifthe observation target is mapped to around the reference white RefW, butthe position to which the observation target is mapped move outward byassociating it with the coordinate (i′, j′) of the point I′J′. Becauseas a result, it is expressed in more greenish colors, coloring of theobservation target can be improved.

Although in the explanation above, at least one of virtual color spacethree primary stimulus values is associated with a point outside thevisible color space, it does not have to be outside the visible colorspace. For example, if by applying, to the coordinate (i, j) of thepoint IJ, an amplification gain in a direction toward the outside of theregion 401, the color-saturation can be improved, it may be associatedwith a point in the visible color space. An amplification gain may beapplied to another primary stimulus value other than the coordinate (i,j) of the point IJ, or an amplification gain may be applied to aplurality of primary stimulus values.

Although in the explanation above, the image processing unit 205associates the primary stimulus value corresponding to the highestspectral intensity with the point KL or the point GH, if the systemmemory 206 stores a color difference discrimination threshold, it mayperform association using this color difference discriminationthreshold. As one example of color difference discrimination thresholds,a discrimination threshold corresponding to a McAdam ellipse can beused.

FIG. 19 is a figure for explaining another example of a correspondencebetween an image-capturing system color space and a display system colorspace. (a) and (c) of FIG. 19 are the same as (a) of FIG. 5. (b) and (d)of FIG. 19 show visible color coordinate system chromaticity diagrams.The horizontal axis is x, and the vertical axis is y. Ellipses in theregion 401 in (b) and (d) of FIG. 19 indicate McAdam ellipses.

In the example of (a) and (b) of FIG. 19, similar to the example of FIG.14, the coordinate (a, b) of the point AB is associated with thecoordinate (g, h) of the point GH. The coordinate (c, d) of the point CDis associated with the coordinate (i, j) of the point IJ. The coordinate(e, f) of the point EF is associated with the coordinate (k, l) of thepoint KL. An observation target Obj1 and an observation target Obj2 aremapped in the same McAdam ellipse. In this case, it is very difficultfor a user to color-discriminate the observation target Obj1 and theobservation target Obj2.

In the example of (c) and (d) of FIG. 19, similar to the example of FIG.15, the coordinate (a, b) of the point AB is associated with thecoordinate (i, j) of the point IJ. The coordinate (c, d) of the point CDis associated with the coordinate (k, l) of the point KL. The coordinate(e, f) of the point EF is associated with the coordinate (g, h) of thepoint GH. The observation target Obj1 and the observation target Obj2are not mapped in the same McAdam ellipse. In this case, it becomes easyfor a user to color-discriminate the observation target Obj1 and theobservation target Obj2.

Based on what is described above, the image processing unit 205 mayalter color allocation if the observation target Obj1 and theobservation target Obj2 are mapped to the same McAdam ellipse. Forexample, it may alter color allocation into the one explained withreference to (c) and (d) of FIG. 19. Thereby, it becomes possible toconvert a color with which color-discrimination is difficult into acolor with which color-discrimination is easy. The color differencediscrimination threshold is not limited to a discrimination thresholdcorresponding to a McAdam ellipse. For example, it may be adiscrimination threshold corresponding to another principle such as thecolor universal design.

Responding to a user manipulation accepted by the manipulating unit 208,the image processing unit 205 may change any one or more of the hue,color-saturation, brightness and gradation characteristic of at leastone piece of pixel data related to a region. For example, data can beconverted into a color that allows easier color-discrimination bychanging the hue.

Although in the explanation above, observation information indicates howvirtual color space three primary stimulus values and visible colorspace three primary stimulus values are related to each other, it may beinformation indicative of the spectrum itself of an observation target.In this case, the image processing unit 205 may determine how to relatevirtual color space three primary stimulus values and visible colorspace three primary stimulus values based on the output characteristicof the image sensor 100 and the observation information. For example, asmentioned above, a primary stimulus value corresponding to the highestspectral intensity may be associated with the point KL or the point GH.

The control unit 201 may generate infrared wavelength information as oneexample of invisible wavelength information for convertingimage-capturing data generated from the output signal of the imagesensor 100 into visible color space image data. As described in detailbelow, the infrared wavelength information is defined based on thesensitivity characteristic which is the output characteristic of theimage sensor 100 to an object light flux wavelength.

The image processing unit 205 may record, in the memory card 220, animage-capturing file in which infrared wavelength information as taginformation is related to image-capturing data having undergone varioustypes of processing. Then, using the infrared wavelength information, itmay convert the infrared wavelength band-based image-capturing data intovisible wavelength band-based image data to generate color image data.

While the embodiments of the present invention have been described, thetechnical scope of the invention is not limited to the above describedembodiments. It is apparent to persons skilled in the art that variousalterations and improvements can be added to the above-describedembodiments. It is also apparent from the scope of the claims that theembodiments added with such alterations or improvements can be includedin the technical scope of the invention.

The operations, procedures, steps, and stages of each process performedby an device, system, program, and method shown in the claims,embodiments, or diagrams can be performed in any order as long as theorder is not indicated by “prior to,” “before,” or the like and as longas the output from a previous process is not used in a later process.Even if the photographing process flow is described using phrases suchas “first” or “next” in the claims, embodiments, or diagrams, it doesnot necessarily mean that the process must be performed in this order.

Various embodiments of the present invention may be described withreference to flowcharts and block diagrams whose blocks may represent(1) steps of processes in which operations are performed or (2) units ofapparatuses responsible for performing operations. Certain steps andunits may be implemented by dedicated circuitry, programmable circuitrysupplied with computer-readable instructions stored on computer-readablemedia, and/or processors supplied with computer-readable instructionsstored on computer-readable media. Dedicated circuitry may includedigital and/or analog hardware circuits and may include integratedcircuits (IC) and/or discrete circuits. Programmable circuitry mayinclude reconfigurable hardware circuits comprising logical AND, OR,XOR, NAND, NOR, and other logical operations, flip-flops, registers,memory elements, etc., such as field-programmable gate arrays (FPGA),programmable logic arrays (PLA), etc.

Computer-readable media may include any tangible device that can storeinstructions for execution by a suitable device, such that thecomputer-readable medium having instructions stored therein comprises anarticle of manufacture including instructions which can be executed tocreate means for performing operations specified in the flowcharts orblock diagrams. Examples of computer-readable media may include anelectronic storage medium, a magnetic storage medium, an optical storagemedium, an electromagnetic storage medium, a semiconductor storagemedium, etc. More specific examples of computer-readable media mayinclude a floppy (registered trademark) disk, a diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an electricallyerasable programmable read-only memory (EEPROM), a static random accessmemory (SRAM), a compact disc read-only memory (CD-ROM), a digitalversatile disk (DVD), a BLU-RAY (registered trademark) disc, a memorystick, an integrated circuit card, etc.

Computer-readable instructions may include assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, JAVA (registeredtrademark), C++, etc., and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages.

Computer-readable instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus, or to programmable circuitry,locally or via a local area network (LAN), wide area network (WAN) suchas the Internet, etc., to execute the computer-readable instructions tocreate means for performing operations specified in the flowcharts orblock diagrams. Examples of processors include computer processors,processing units, microprocessors, digital signal processors,controllers, microcontrollers, etc.

What is claimed is:
 1. An image-capturing device comprising: an imagesensor that is photosensitive to light in an invisible band; acontroller configured to generate invisible wavelength informationdefined based on a sensitivity characteristic, which is an outputcharacteristic of the image sensor, to an object light flux wavelengthfor conversion of image-capturing data generated from an output signalof the image sensor into visible color space image data, the invisiblewavelength information defining a virtual color space; and an imageprocessor configured to relate the invisible wavelength information thatdefines the virtual color space to the image-capturing data.
 2. Theimage-capturing device according to claim 1, comprising a plurality ofbandpass filters that allow passage therethrough of mutually differentparts in the continuous invisible band in the object light flux, whereinthe controller is further configured to generate the invisiblewavelength information defined based on the output characteristicdetermined from: wavelength bands in the invisible band that theplurality of bandpass filters allow passage therethrough; and thephotosensitivity of the image sensor.
 3. The image-capturing deviceaccording to claim 2, wherein the invisible wavelength informationincludes a plurality of invisible primary stimulus values determinedfrom: respective peak wavelengths of the plurality of bandpass filtersand half widths corresponding to the respective peak wavelengths; andthe photosensitivity of the image sensor.
 4. The image-capturing deviceaccording to claim 3, wherein the plurality of invisible primarystimulus values are associated with information related to mutuallydifferent regions of the visible color space.
 5. The image-capturingdevice according to claim 4, wherein the invisible wavelengthinformation includes information for associating the plurality ofinvisible primary stimulus values with visible primary stimulus valuesin the visible color space.
 6. The image-capturing device according toclaim 2, wherein the controller is further configured to generate, asthe invisible wavelength information, information about a lower limitsensitivity and an upper limit sensitivity of the sensitivitycharacteristic in the continuous invisible band.
 7. The image-capturingdevice according to claim 6, wherein the lower limit sensitivity isassociated with a coordinate of a blue region in the visible colorspace, and the upper limit sensitivity is associated with a coordinateof a red region in the visible color space.
 8. The image-capturingdevice according to claim 2, wherein the invisible wavelengthinformation includes a reference value associated with a particularcoordinate in the visible color space.
 9. The image-capturing deviceaccording to claim 8, wherein the particular coordinate is a whitereference coordinate.
 10. The image-capturing device according to claim8, wherein the reference value is a value within an image-capturingsystem color space.
 11. The image-capturing device according to claim 8,wherein the plurality of bandpass filters are three or more bandpassfilters, and the reference value is determined based on respective peakwavelengths of the three or more bandpass filters.
 12. Theimage-capturing device according to claim 8, wherein the controller isfurther configured to set the reference value based on a spectrum of anobject an image of which is captured by the image sensor.
 13. Theimage-capturing device according to claim 12, wherein the invisiblewavelength information includes information indicative of a gain bywhich an output signal of the image sensor is multiplied.
 14. Theimage-capturing device according to claim 2, wherein the invisiblewavelength information includes information indicative of a color gamutin which an image based on the image data is to be displayed.
 15. A datagenerating device comprising: an image processor configured to acquireimage-capturing data an image of which has been captured by receivinginvisible band light; and a controller configured to generate invisiblewavelength information defined based on a sensitivity characteristic,which is an output characteristic of an image sensor, to an object lightflux wavelength for conversion of the image-capturing data into visiblecolor space image data, the invisible wavelength information defining avirtual color space; wherein the image processor is further configuredto relate the invisible wavelength information that defines the virtualcolor space to the image-capturing data.
 16. An image processing devicecomprising: an image processor; and a memory having computer-readableinstructions stored thereon that, when executed by the image processor,perform operations including acquiring invisible image-capturing dataincluding at least three types of image-capturing elements correspondingto respective wavelength bands in an invisible band passage of which hasbeen allowed, acquiring observation information which is informationabout an observation target, and relating mutually different regions ina visible color space to the respective image-capturing elements basedon the observation information to generate color image data.
 17. Theimage processing device according to claim 16, wherein the operationsfurther include acquiring information related to a spectrum of theobservation target as the observation information.
 18. The imageprocessing device according to claim 16, wherein the operations furtherinclude acquiring characteristics information indicative of asensitivity characteristic, which is an output characteristic of animage sensor that has generated the invisible image-capturing data, toan object light flux wavelength, and relating the regions to therespective image-capturing elements based on the characteristicsinformation.
 19. The image processing device according to claim 16,wherein the operations further include relating the regions to therespective image-capturing elements based on a color differencediscrimination threshold.
 20. The image processing device according toclaim 16, wherein the operations further comprise associating a regionoutside the visible color space with at least one of the image-capturingelements.